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\ ' */ A hew role f or . evaluators is propo&ed, ?^l>.icH 

reconstruct^ 'thf ^valuator Into.an infcrmaticn specia^-ist, whb' is part 
of .a projep t team ehqaqed.^^n self evaluation. JEhe evaj^iatcr' ^ rcle^ 
has 'traditionally been complicated ty; disagree^ments atcut whit is 
expected from the evaluator ; ^ By dS^^^^ng^- perce ptions about t^e 
eveiiuator's power to effect change; EyM^^ influepce of personal 
values and ra-etbodo'loqical preferences ;^d by the conflict between 
internal reports o.f' findi|ic[_s (to be used prcgr'air- imprc vemept) , . 

and external reports (to be -Hjsed to aake^ midgments regar ding^pr o^fram 
accountability). The proposed role of/ the*^. i^f orpiation sp€ciali^;t 
/iWludes five major functions: (1) program clarif ication-^askin^g 
questions abdut the evaluation plans ard. criteria for assessment ; (2) 
(information structuring-- developing ferns ard tests; (3) information^ 
acqursi^iohr-collecting data; f4) inforroaticn proTisicnrr- analyzing 
data and pres.entinq results,; and (5) technic^dl assistance— giving 
advice in his, or her -areas of expertise^ ccrductihg pilot tests, or 
seaxchiriq the ^^i te rature. Self evaluation (hy the project staff) ?.s 
empba^ized a.s a^ means of increasing the likelihcod ^ of program . \ 
improvement. Suggestions reqardinq the acccunta-kility r cl€ cf \ 
external: audits are included. (GDC ) . ' \ 
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A NEW ROLE 



f4 EVALUATORS'^ 



C. Alkin 



■ . Before 1 begin my speech, I vf Id like to ask the audience to, not applaud 
w:ildl.y, to not stdiiip your feet or ottierwise show indecent appreciation of| the 
first sentence which I am about to state. Tjie modest pYoposal that I bring to 
you today concerns a nevi ro]_e for the eva lua i that^ new- roTe has been f . 
gradual 1y evolving over the years ; the new role ttiat I propose for eva1u;ators 
is. obsolescence ;, in short, I propose tha t we do away wi th evaluators . | 
/ } If yoU' think about it7 this is really a very sensi b1 e^Ag£|es ti on . lAfter 
all, what are evaluators?' Why do we need them? Of what use, are they? i 

' ■ " ' ' ^ h 

Perhaps the theoreti cal ' consideration of these questions^is one of the most 
difficult tasks that we might undertake, because, eval-uajor^ are real lyf defined 
by the roVes 3nd functions that, they^ perform and these roles and funcljlions 
vary. Certainly there is no complete agreement , the literature of 0ur field 
as to the proper funrtions of evaluators. ^ There are aspects of the d(Lfinition 
on which all agree, but it is in the interpretations of^ these broad beliefs 
that disagreements, like hot air, begin to surface, fo illustrate the 
complexities of these issues, Vetv^e present "some exairiples. 



l uncheon speech presented at the Fourtli Annual Conference and 

Exhib i tion- on Measurement and E valuation, office .of ^he Los Angeles ■ 

County Sui^nrinti^^ndent of schools, Mr^rrh , 1Q77. < ' \ , 



I. • ATI agree .that the evaj^aton jsj^spUnsiljle for'thc collection 

.of infomation (whether or not .he -or she personally collects •• 

it). _ , , . ; ^ ■ 

A. Some maintain that the most appropriate type of infor- , , 

mat ion is forma], test data. - • 

B. Others maintain that the leas_t ajyjj^^oprj^at e type of infor- 
mation is formaT test data. 

II. All agree, that "valuing''" is a part of the evaluation proces-s. 
-A. Some^ maintain that -it is the role of the evafuator. to , 

value. ^ ^ " ■ • 

B. Others maintain that tf is the role of the evaluator to 
assist decision makers in clarifying their' values. 

III. All agree that evaluation .takes place within a political 
setting. 

A. ^ Some maintain that the evaluator mu^t be pure of heart 

■ (mind, soul, and hody^) and simply present his fi.ndings 
and not be influenced by the political system.^ 

B, ^then^ maintain that the evaluators must be cognizant of 

and w.oV^within the political systjem. 

IV. All agree tha|^evaluators^should look at the outcomes of 
programs , 

A Some maint^iin that the outcomes examined should be related 
\ ■ ^ 

to the objecfives proposed. / 

B, Others maintain tha^he outcomes should be examined 
without prior knowledge or objectives. 

V. All ^r^e that evaluators must have exportise. 

A Sdhip nidintain that tecfmicai sof^h i s t h j H r-r ^ hV' 
' y hallmark of pxpertise id an o'vmm^-' 



B. Others maintain that expertise is instead an understanding 
of instructional programs and organi ^rational dynamics. 
VI. An agree that evaluation reporting should occur (and that's 
about all we can say that all agree Dn). 

Some maintain that reporting should be a complete, unbiased 
technical .presentation of facts, 

B. Others maintain that reporting shouTd be from a multiple 
source orv an advocacy basis. 

C. ■ Sbme maintain that written reporting is the most important .^^^ 

D. ' Others maintain /that verbal/informal reporting is the ^ 

most simportant. , \ 

The theoretical, literature on the role of the evaluator presents a' 

vast diversity of views which rDay make it a fruitless effort to discuss • . 

this changing role ba^ed on any ideal description. Let us instead 

retrea-t from the ideal "theoretical evaluator" to a view of the "common 

man" evaluafor as he actually functions (and non-functions) in the real 

world, with all his strengths, v^/eaknesses, desires, -and hahgups. I'm 

afraid that with that last addenda, I've bitten off more than I can 

.chew, 'but lefs at least take a crack at beginning the examina^on of^ 

the evaluator. To do this, let- us first consider the e/tent to which 

power Js an^ influence in the. role of the evaluator., - 

I ' . - ' ' 

EVALUATQRS AND POWER 

y ■ 

First of dll, evaluators are people, and their performance .as 

people IS siibjer.t. U^ a-l I ot the i^j 1 es. t h^j ^. ^/.e hiow juvef n i rw j' hui^Hi 
behavior. Wh i I e r.h ink i ng over this section ^'f \Mv S[)(MMh. 1 w<is a 1. 1 r-i-p t i ng 



to find an old adage (are there such,:th'ing^ as "new adages?",) which 
Siight fit this current situatjip^rf. /The first thoMght to-^come to my mind 
was the' saying "power". corrupts; absolute power corrupts absolutely:'"' 
The role of the evaluator is sometimes 'aki n to the situation just rJescribed. 
(The evaluator-by virt^je of his/her position is placecl in a position 
t*hat carries enormous potential fgiK power--progr:ams wi^l be funded, 
increased, decreased, personnel will be hired and fired,' administrators 



will grbw in importance as their administrative domain increases or 

■ • <^ " ./ > 

•alternatively have their power fade as the .size of their domain contracts.) 

Sometimes, however, it doesn't work" out that way and a- more a'ppropriate 
adage prescribi;ig the evaluator's role might be- "lack of pow^ when you 
^thought you had it>or ought to hav-e had it, frustrates." .(Aha, a fiew 
adage)". Sometimes, the evaluator only thinks he has "pdwer and in his 
behavior acts under the assumption that people are ^i?tening and will in^ 
some wdy'>nitiate action and make decisions based upori^Be evaluation 
^>eport. Such stuff is . the* fodder, for frustration. ^ . - ^ 

The point to be made of all 6i this is that evaluators are viewed 
as having power, or potentially having pow^r (with this view sometimeb . 
only being held only by the evaluator himself k Power! Does t^^ valu^' 
really view his role as one of power? Surely, the presuniptlon on the 

■ - r * • . 

□art of evaluators that'their recommendations* wi H be acted upon that 

. ; \ , ^ . 

the results of their evaluation findings^;/. , be ^ )rporau^cJ i. the 

" \ 

decision process", surely the disappointment by evaluators in instance's 
^ where their reports do not have greater impact upon decision making are. 
"all manifestations of the anticipation and expectation that the evaluator 

is in a power posi t ion. 

One. possible way of changinq the role of evaliiatoys i^ -the r entoval 



o.f^ower, actual or perceived. Now what are (the imp l^i cations of this 
statement in terms of the role and funcfibn of evaluators? Let me 
describe t^ee of -them^^^^^fTt valuing responsibility, the reporting^ 
res'ponsibi.l ity, 3) the technical respop^sibi 1 ity. . 



The Valuing Responsibility 

^s previously noted, there is no complete agreement on the extent 
to which the evalujator 'has the responsibility for making independent 
'e^jaluaUive judgments on "\he data and information col.lected. There are^ 
some- who feel that as an independent expert, it is his responsibil ity to , 
bring his own ya'lues and judVents to play in interpreting ^ata. There 
are' Others, I . among them, who maintain that the evaluators role is 
helping ta clarify the/value judgments and>value bases of those responsible 
.for operating the program^, participating in it, or in other ways affected 
by It. " ^ ' . V ' 



The Ale definition that accords the ^eval uator the responsibility 
oTTnvoking his personal value system to choose which data will be* 
collected a^out programs is a great source of power (actual or perceived) . • 
Som^^times' |h%^ other aspects of the evaluator*s role, including the 
context in which he finds himself, are sufficiently forceful that they 
allqw the evaluatt^r's val ues'--i .e. his/her choice of what to look at— to 
be accepted and the cdncomitan^t results and rexommendatibns of the » 
evdluatipn^to' be accepted. More frequently, though, the valuing activities 
of tKe evaluator form the nla.jj>r"^basi s for dispute of evaluation reports 
and subsequent non-attention to and inaction on the findings of the 



f^va 1 ua tion . 



One thinq is. clear, and that is if we are f.o reiitovr tl>e pc^wor 



perception of power from the h£nds of the evalyator^ clearly both aspects 
of the valuing function must be removed as legitimate roles of e\(aluators. 

The Reporting Function , ' ' 

' ■ . ' 
While one of '4he major sdurces of power of the evaluator" is the • 

vaKling function, perhaps an even greater source of power (or perceived' " 

pow?r) emanates frpm the reporting funct^ion. The^ evaluator, after all, 

■is in many. ways anal agous to the school ^inspectors of many European 

s^ool systems, .these people go around from s<Shool_ to'l^ool in various 

"Mparts of the system and "inspect." . In jessence their function is to make 

evaliattve judgfiierits about the school and it's quality in tenfis of 

prescribed written standards and in many instances in terms of their own' J 

conception (their own va-lue judgments) about what a quality school! 

should look like. vJhese are positions of power held by people in essence 

performing eva|uativ\ functions. The power in large part derives from* 

the^responsibility to report. This responsibility always relates to 

another agency, or to a higher level of government, 'where- action at the 

higher levels or other' agencies may be taken bas^d upon the evaluation 

information reported. It is cleaVly an instance of evaluator powe^. On 

the continuum .previously described,^ra;nging from Corruption to frustration 

■I would venture to say that the traditional school ins^ctor is far more 

• toward -the corruption end of the continuum than the frusVation 6nd. 

A similar kind of reporting role exists in many of the functions 

- that evaluators perform. in this country tcfd^.". Either,, they are r^orting 

to another agency with the liklihood that subsequent funding might be 

' witheld (aUhough we' know that this does not usually hap^n) or 1;he 

evaluator is reporting to a liigh level of the school 'district's ornanizati 



witK all; individuals and participants administratively located below 

that level considered as'{fair game" for his evaluative judgment, critiques, 

kudos, and outftgtit criti^isrps. 

-Sometimes, .•evaljjators are asked to occupy a dual, role, a ^role as 

Wember of the project staff who^rovides' information towards internal " , 



did 



U 

program modifications during program operation and simultaneously t\]e 

"7" .V . / ' ' - , •/ ' 

role of externa.l judge, valuer, and v'isporter. -In a ni\)nograph that- I 

^several years" ago, (Evaluation and Dec1-sion ^MakincL: The Titl-e VH 

Experience ), I discussed this problem in terms of the role ambiguity and 

conflict'^^that Title Vfl evalu^itors face in -their 4ual respon^ibi 1 i.ty, of 

meeting"^ local information needs .and reporting "unbiased" resufts to t^e 

federal agency. ■ In- that study, my col leagues^and I found that the ^ . 

~ • ' • s .... 

•evaluation results had -no impact upon. federal decisions related to 

projects.' We' also fo\jnd that those evaluation situations in which the 

Jocal deciffion makers viewed the evaluation as being usefuTand hav.ihg 
. > J » P • ■ - ^ . 

-impact upon modifications in program tended to, be highly related.t^o^ 
^•s'ituatic^ns where evaluators had apparently abrogated (or made light of) 
^|heir federaT reporting responsibility and had instead focysed most 
heavily on their within project -eval uation role. ~ 

Why do evaluators need to have the e)*terna1 reporting responsibility 
at all? I suppose we could think or -two^ossible functions^ evaluation 
reporting: 1) prjDgram^ improvement, and 2) accountability. Let's look 
-for a mojnent at the program improvement question. The literature of 
ey^'Tuation is , replete with articles decrying the non-uti 1 i zati on of 
evaluation information in progranr' dec i s ion making. Usual ly' thes.^ articles 
dre written by evaluators whose expectations nf ()ower and influence led , 
theni to beMev'e that people would pay heed to their coqent advice. -Tlir 



scho6l adurffilf trato'rs are surprisii^gly eager to' echo the cry of the 



frustratifcd'^valuators that evaluation niake£ no ditflience^. (Thinking 

perhaps that if we've got those guys--the evaluators--on th6 ^,un, why ^ 
' . ■ /- -. 

keep, them thene? Maybe it would"be ,h6l pful to get theoi^off our 

^ ^ ' ' ') ' ' ' . ' • ^ 

.bicksO If eva^aatibn reporting as it is/currently structured has such 
small impact on prograifP decisions and_^ program improvement, .then of what, 
use is it? . ■ . . 

[ The second possible reason for evaluation reporting is that of 
accountability. It i^' clear that evaluator' s .as^heir role is currently 

■ defined, are^up^Me to fulfill simul taneously 'th\ "information source of 
p.rogram*improvement" role find th'e "agent of the forces of accountability" 

• role. ' The accountability roTe acts as'-^n impedjment tq, the program -■ 
improvement role because' accountabil ity candies with i.t power or 'the 
presumption of power. I'believe that pt;c>gr3m i(tiprove|nent thrives best- ^ 
where informatioh flow is non^thfjeatening; I believe that program - 
improvement thrives best when the agent' (responsible for reportflng is 

■ different, front the agent providing information ^Tnd counseling. We have 
J^ng recpgnized this principle |fT publ ic^ high^school s' by separating the 

' functions of attendance officer or Dean of Students from the role of ; , 
class counselor; J3ne -person has the responsibi.l ity of report^ing, , monitor- 
ing /obtaining compliance; the other has the role of counsel ing, -^ggregat- 
ing informatian, providing data--X)n colleges, on jobs, e-tc. • ' , 

It is'clear to me th'at at the-'hear.t of ,eval uation i s 'the pVovision-' ■ 
of information for action,^, for decision- making, for program improve^ient . 
To attain that^position, I bel.ieve that the reporting f uncti on , ,the 

"^source of pat.ential thre'atening power, needs to be deleted from the 



pvaluativp role. h \ 
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- Asidf* from t.ho pov/er thtit ^iccruo^ 1.o (.-y ti I t.ors in tiu^u; tiiru.rfoTi 
d'. valuers .and iir rolationstnp lo^.lWMr r^^^^^'^ f^fn^(H)n, a source of 




power exists in th? ^^chi^ic^ 



(lai crc^. Many iMui-^ the . ^ 
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external agency or \o higher ech^ons within the same administrative 
organ] zatjpn; and we do not need to have evaluators have free reign on 
the use O/f their technical expertise. If vie do away v/ith all of these 
things perhaps its best^ to conceive of . a new ro1j for evaluators-- 
obsolescei^cG. For if d^re to' d>}T i iu\ a n^w .evaluative role free of 
pov;ar, or tha appearance -^'Z pov;.;:r, or ev^^n oi so i r-p3rc:^P': lons of pow 
then v^hy car / vn ch it ^^\y^ ::ii':o'C)Oi> tinqeci -.ici::: u" "O'/^' i -ictor , ' -.Like 
the cultural baqqacio chrr]e(\ by i:ho nov; irnniigranir: i:o this country who 
(/^ :an:ci Arnori cam /:i3c: , perhaps j3 too can di scarT' tfuroMirtanguB-gG r""a^^ 
a new naui'i. anf'-leL clie title "evaluator" quioUy fade into tlje past. 

And nov/ lot us consider the steps necessary ror achieving the 
obsolescent state for c/uluators.^ As ^v/e gradually phase out<'evaluators 
- (v/hat on oai-th am ^' doing? Con^t I know that I ceacli in *a training 
r)rogr'aiii for jdncafional - I oa i:ors? ) flonotiiol ess , lot mc continue. (I 
say ciris confidontiy !'nov;iug fuH woil ch'o naiuro of my speculative 
ideas and trie cibsolure unlikiihoofl era;: the bureaucratic system could 
act on these suggireLion .. ar/ less time than tlie or ?b years I have 
' before^ r-i. i rriii^^n ( .it Uie university.) lo eontinue, h't us consider the 
creation oi ci fiew [)osif:iorU a new proi <,^ss iona I [)osirion in which trie 
rjccupant do.-, noi iiave t[ie respons i !) i 1 i ty Um (^:;"tab 1 i sh i ng personal or 
ex terrui 1 va i t! • ■ i^ri^ i'r^rr^ru^es in judf)in(] data.^'^.et qs -c'sCablisfi a new role 
in which lUi' o^:cu\h\nt (ioe^ not have lIc^^ resi.ons rl)i 1 i f y for rei)orting ij; 
\ oth(^r a(jenci(!'> or r.o othf^r levc'ls withir; thr saitK^ <t(j(Mi(y. frO. us t^stablish 
a uew pro t f'^' s i final pn-.ition in "jhirii i h^^ individu<{l i ( ti s^UcUj^.ul t ronf 
■ir,i?irj hi'. ((M.linoai (>q)(u-tise fi , a iiumm'. .iMainirK) iir, own d*", 1 1 rU 

/ lU 
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\' and preferred preferences on program operation,- b,ut instead recognizes 

the necessity af 'sometime settl,i,ng for\ less : rfjan lechnicarexcel 1 ence. 

Let's call this new role/f unction/porson an infurmdtion specialist, and 

consider hijii or her ^s a member' of the projoct or. program team, responsible 

for satisfying the i nf oniia ti on needs of the ovo^ed/. 

Uhai: v;oiii,i Vne in forfna c io;i Sj)i-?c i c; 1 i s 1. oo?' Gmu I'Moor ld^"\t vi-jnC':io;i 

is tha- of proqV-din c.lari f ier. As ^o. i n'd i vid'ia 1 . rospons i e for-inaking 

\!jc]qe-^ L io/is abolii: (,he ii:3ans oT n;^::dSur ing ci.o^i ; niue/i i: of program 

0pTsclTVes"'anir'"r"0'!rpDTis"T^ 

so.lijciori ';if ■Legies of a pro(jr:.\ni have been irnp I emeu ted , Uie important - 

role of program clarifier emerges. Perhaps you might think tf^at as v;e 

^ have described' it, the appropriate description for the role is "devils 

advocate" 1)111 rt goes far bayond tfiac. In his roio as program ciarifior- 

the information special Isl v/i i ! ask questions such as: "Uhat are v/e 

f^oally tv^yi;j(^ i.o do?" "Do we 'jo\]t to be ii.-:! d ' accountable for that?" 

"iiov; /il! ' /loi^y vjhcM: v;.: ' v in is?" or, n\or?! nonerdlly, "Hov/vni'l '-/e 

knovi v/hci ciief . . . ?/' "llhai do wa ic-ran by..,?" "Vina'. r^^c l:^.? is ti cs v;i"l k 

/ - 

Ls' [)rese/;t in order to observe..,?" And, perhaps chc mosL important ■ 

f|ijesLlOn of ail "do v/e really pl^^n to do that?" 

A st.MUHi tunL.^ion of t()e inforvnation spaci.tlr.i is that of informatjon 

s true tur 1 rir^ ^ fhis person would t)e resp'ons i 1 e irrr drvisinq the fornts, 

S'-^ b^., t. i fuj Lh;- i.(^SLS and cons tr u(, c i nq ' o thoi' i /is i.r-udien » ^ lor the accjuisition 

^'()i,dat,a; .ill \(> Ui' [ir^esented to (rr{\]eot t.n II (M()ue^"^ roi' thc'ir acceptance 

'Hid (jgref-ncf ; i -i > i.n -./lieilK-r' tho i ns ^ rument^-. -i[>fusc Mi;' i^'.sorKO of -what 

v/'is ifUenood. ' 
I 

A (. h M M I iitK i : i j/i o i (Jio J ii ( oriii.! i o wi 'auM i a 1 i ■■ i.lid t, u i i n t oriii(i 1 1 on • 

n 

•J ( ' MM s 1 t I Mf) . ]\] ) \ I m 1 (j(lo', iiKMU t Mr M)(] o 1 pn H I f ,1111'. i fi ,u < nr<l(in('r w i f-P 



agreed upon' instruments in order to gatl^r systematic data on the -extent ^ 

to which the programs have been iinpl eiiiented in the pre-determi/ied manner 

' ' ' ■ ■ '■ . 

or what they look li4ce--plan or no plan. Some imormat ion to" be acciuired- 

is.^of 'a'more foriiia.l variety arid the acquiA-iLion proct-ss entails ttu' 
necessity of administering foniial, tests. Al l of this -is done witii the 
concurrence and prior agrsement of fellov, project member's , because this 
is ::ha agreed upon function for Ghe infomiai,ion special isi,. 

A further activity for this person is hs uifmnAtio£ Piio^djjr • In 
■ this role the specialist may hav,- co perform analysis of certain of the 
data collected, prepare charts, tables, or in other ways clarrldy'the 
meaning of the results. (Soma information, for example, the program 
implementation monitoring data, vjould undoubtedly be provided in a more 
' informal manner to "project staff during thc.'^!;ourse of the pro ject- -such 
as during staff meetings/) . 

, A final role of the infonnation specialist is as te;cjinicaj_ assis tant . 
!n this rol^ the infonnation specialist brings his technical expertise 
in the ar.^as or research Sesicjn, statistics, measurement, etc. to beav;^ 
on the problems and requirements of the project. The specialist may 
also do literature searches, pilot tests, etc. The information specialist 
I's A resource ()ersnn available to tell his fellow staff members what i ^ 
technically most correct but recognizing the poss i b rl i ( ^ f^^eed prot)^abilit 
that concerns over program relevance, concerns over i reformat i o^i relevance 
for decision makm*] ran. supercede technical considerations. 

In this [oodel of pr^ogram evaluation, values are determined by the 
administrative stafl, or ^t^re ferabl y by the totdl project [)ersonnel-. In 
ttiis liiodpl, rei)0rtin() (jf [project attainment, 'sufce.'.s. iikkI i f i ( a t i oris . 
Pt(^^> rlone Xri the [)rojer t administrator ()r s ba t i '. (For <\\\v\ all. wc 
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are far more concerned about withi.. 'taff initia^/ni program change based 
upon systematically gather^ rmation than we are with external 

reporting of project performance based upon reporijng formats devised 
elsewhere and generally not f6und to be meaningful to project personnel,^- 

K ' ' 

Beyond Obsol escanca - / •. 

Now that I hav3 succeeded m 'creating ayYlodel vjhich niak^s che 
evr:luai:or role obsolete, let me consvder the implications of this 
acLion. I have examined the role of che inrormation specialist. lf> 
es^;.enre, this role places the emphas^s^ on 'project stH f-eval uaci^^^^ 

• . .- / 

reconstructs the evaluator role inio an information specialist who is a 
part of the project team v;hich j's engaged in sel f-evya luation . I would 
fiiaintain that, this is a scanct^ more consistent vfith the liklihood of 
attaining program change. ' 

] recently caifie acv^oss a boo!: entitled EnhancjjricLj^^ by 
^M( har(l DeCharrns ; Wiiile his iocMs \s ^Ihe classroom, .the teacher, and , 
students, nonethelc^ss the analogy or ^the point he makes seems to me niosc 
.-ippro;i)riate in the consideration of a v^edefined roie for evaluators. 
DeCharms maintains that the greate*>t motivation is attained by students 
when they function a^s theiv; own evaluators-~when self-evaluation takes 

place. - ■ ^ V 

In fart., sol f -evaluation occurs in all- situations. We -are constantly 
ciwaro, as individuals, of our status, or our achievements,, of our 

'iftainmenf s find we Dlace value judgments on Lhe extent\to which we 
' "<P ' • -f > 

consider i.hose alLaininents satisfactory. When a student in a r, fas sro.om 

gets an BH, on ,i L<-arher made test, he may vif^w t.h.u. as either a ma-inifice 

■ I'MiPvemont or- .) inisiM-at^le failur-r* de[)(»ndant uf^on i) i s Dwn v<iVijp t ysu.K^v/O)-^ 



1 conception- 0/ who' hf what he is capable of, and probably 

;s own aspirations as vk' ' 
To some extent the v/ay in which a student views* a piece of evaluative 
data "(such as a test score or % correct on a test), is 'set for him , by 
the evaluative statements made b| the teacher. Thus when the teacher 
says ■"over' 85^;', is an A on this tast," students will tend to view the 
daca item in a more favorable li^ght. To some extant the student is 
removed from active participation in making a judgment about the extent 
to which the performance was .satisfactory becapse the valuing has already 

■been" done for h i m . BeGha rms - mo i nta ias'. tha't . gre^^.te r . .s tjden t j n yo ] vemen t ^ 

in the valuing process, the establishment of one's own goals, and the 
determination of standards of • expectation lead to greater committment by 
the student and greater motivation. ^ 

It seems to me that this is analogous to the situation we face in 

school districts where programs are being evaluated. Let us consider^ 

y 

tor example, compensatory educafion program;:, in two hypothetical California 
^^'nools.. (Now you better listen carefully, because I'm going to test 
you on it in a few minuses.) In the Webster school, an^external^evaluator 
has been hired to provide the evaluation of tKe program. He came on the 

;seene in pferhaps September or October, was provided with the A127 plan 
:fc# the school and was as!c^ de an^'ndication of the evaluation 

M!24||jf'"tVthat would" ' '-.'^d during the cou^e of the school year. 

"^M^^^^^^^^ program plan has a section in wh^ich evaluation instruments 

^ '-are defined,' but 'due tO'previous bad advice, many of these are inappropriate 

'i . . 

for the objectives. Moreover there are a number of places within the.. 
plan which indicate that there will be a site developed instrument. The ^ 
oval'uator c:o(nes in, he's asked' by ^project personnel to provide the 
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necessary advice. /He accepts the plan at face value and makes/some 
recomnendations as to instruments that might be^ed.; In several cases 
inwhichhe feels no existing instrument is appropriate, he devises a 
measure for use'by the project per4nnel . (Perhaps this measure is for 
an objective in" one of the more amorphous areas, 1 i Ice paren/ partici pa- • 
'tion or health auxiliary.) The project evaluatpr then conducts system- 
atic, observations in the classrooms, and meets wUh school personnel in 
order to conduct the process evaluation. He views things going on in 
the classro9m, he notes problem areas; he recognizes needs that are not 
^be-ing met, and writes a process evaluation report to the school, certain- 
ly with a copy to the d'istrict offi.ce; Tfiis report not only comments on 
the extent to which the stated solution strategies were implemented, but 
in some. instances points out probl em^ areas that, the evaluator feels are 
appropriate to menti on-:. | perhaps these v^ere solution strategies not \i 
listed in J:he plan bi\t which seemed to make sense, perhaps they v/ere 
aspects of a program that tfi'e evaluator feels are a part of "good" ^ 
education (e.g., sufficient individualization). The report is received 
by program personnel who for the most part are not philosophic?' 
committed to what the evaluator ha? i to |ay. The notions 
are hi^ values, the standards of acceptability have been estabi 
a level beyond their involvement. Naturally they do not want to 
antagonize' him (primarily because he will be writing a final rep^ .a 
the end of the year). Naturally they do not want to look bad in the 
eyes of the district offic^ "br to present a negative image to their 
- schooradvisory counci 1 . " Thus , ^.he principal, the project administra- 
tor, if one exists, and to a lesser extent other profess i onal. personnel 
in the school nod their collective heads affirmatively cind pay 
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lip service to the implementation of the suggestions presented in th-e 
process, evaluation report. The moiie of the day is Gompl iance^, but 
-compliance is not comnittment. 

Continuing on with our -safla, our evaluator proceeds through the 
year, considers the evaluation tests that are scheduled ta be given aear 
the end of the academic year, and injsome instances he .notes measures 
that he feels are inappropriate fovynieasuring the objective. In other . 
instances he's not toU]^^ with the measure that's been selected 

in terms of its comprqhensi v^ss and in some cases he just simply feels 
there are aspects of progr/am .operation that it i^ important to know ^ 
sfc^ething about that Bre>^d^t ref^l^cted in the measures that haVe been . 
specified in the A127 form. And so he presents a' 1 ist of addi ti onal 
measures to be collected^ ^^ditional instruments to be administered near 
^the end oj the academic year. The data collection takes place, the 
evaluator does his^ analyses and in addition to preparii^"" the evaluation ' 
report for the State, h^ prepares a surrr-^^' ' ^--oort ,;for the 

school and the district. Th-^ rop . and for 

.nost part they don't unci-rstand the jres or why they were selected 
or vihat they have to do with their prog^ - ... and besides .that its pretty 
near the end of the year and there isn't too much- that any one can say 
right now. So the report gets filed away in one of the drawers in the 
principals desk' and everyone says, "Gee its sure nice that vacation time 
is here. You know maybe v^fe ought to do something about improving our ^ 
program. We don't quite know what but let's put that off and we'll talk 
about it next September." 

Consider a second 'Schox)ll , ^the Merriam school. Let's assume that at 
this school an- information specialist is appointed instead of an evaluator 



NOW, you have to make the decision as to whether you want to appoint 
someone from within the schofrl to this role, .perhaps someone -from the 
district office, or possibly even an o'utside con^tan.t who you might 
previously have hired as an evaluator. You think about it. Which' one 
of these do you vjant to hire? Which of these do. you .want to appoint as 
information specialist for the Msrriam school. ^ Pick oine. First, we 'IV) 
Xd to- be sure that vjhoever you picjced is aware of and understaiiis the ' 
function of an information specialist. Then, we'll have to be sure that, 
that person has the appropriate technical skills. • , 

Now, here is the test- that I promised you a few moments ago. Think 
aboJr^he activUies that went on at the Webster school and consider the 
role and function of the .information specialist as I described them to 
you earlier. (Program clarifier ,^^Jn^AHpation acquirer, information 

structur.er, informatioi :der, technical assise. / I want each of 

you to think for a moment and write 'the next minute of my speech in your^ 
own mind describing how the information specialist, vjhether project 
internal, district or external, vjhichever you happened to select, how 
the information specialist Vv'ould operate in facilitating self- 
evaluation. 

Now, let's hear from some of y®u as to what your information 
Specialist did (facsimile comments from audien^ce including clarification 
of goals and objectives of program, more precise and understandable 
statement and solution' strategies, measurement instruments desj^ed by 
fi^-oject staff per technical assistance from information specialist, 
collection of data, analysis of data, discussion by projeclj^ staff of the 
meaning of the results, with technical assistance from informapon 
specialists so that group might write the evaluation report-* Istb specia 
pcrnii ss ion of SDL ) . •. • ' 
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Some Loose Ends > . ' ' • ' ^ 

All of- this hypothetical discussion, the cr^ate/obsolesdjnce of 
the evaluator and tlie .definition of a new professional posit/on as . .. 
information specialist, I am sure has lett som|^loose ends. I will 
attend to some of these, but first let me briefly summarize. Earlier in^- 
this discussion.! focused on the need to de\yise a betted' mechaaism for 
providing inf ormatT'on for program improvemen^f. I maintained .that this ' . 
.is the proper artd appropriate role for evaluation and that perfiaps the 
ambigui^ty assocjated vjith functions of the evaluator, the 'pi^oblems ■ 
associated with evaluator power, the appearance of ^power, or the self- 
perception of power, ?.s wal ; as '..i-^ bOir,av;hc . ainted image of gjyal uators , 

■ all made it beneficial to do away with eval uators and' create, a new 

position for the person performing some of tj;ie functions of providing 

■ ■ , ■ 

information for program self-lv=aluation . I'd also noted that another 

apparent function of evaluation, in addition to the provision of iriformation 

for program improvement, is the provision of information for accountability 

■ purposes. 'Vt vjas my view that ft? was simply not possible for the same 
individual to fulfill both functions simultaneously. In fact, I pointed, 
to a study th^t I have completed that demonstrated this parti'cth^ar role 
ambiguity and' di scussed some of the implications of it. 

Thus, in focusing on the function of providing information for 
program improvement, for program decision making, we .tiave apparently 
1eft°an enormous void related to the evaluator' s function of accou^- 
■ abilit)^!^rm rjiot so sure that such a void really exists.) With evaluation 
defined as emphasizing self-e\^luation and en}pl oyi n^/he services of an 
information^specialist, it would seem to me that the external audit 
function would grow in importance. The people in the. Research and 

18 ■ ' ■ 

ERIC 



Evaluation Uni;t of Los Ang^el.es County schoojs, who havQ pioneered the 

broad scale i^pl ementatian of audit procedures, know that the conduct of 

an audit and the conduct of- an evaluation are quite different- functions. 
The amount o'f time consumed i^rTtypical evaluations asj we now know them 

^ su'bstantial'ly ' gv^eat^r tl^an what is required in '^^sfeidit and a good 

dea'^ less appropriate iio the .accountabil ity function. ^ Evaluation, even' 

vihen done best is not as appropriate for the determination Bf compliajnpe, 

and the verif ication^*©^ results as is the audit procedure. Thus, the ' 

initiation of self-evaluation procedures and the changing role of the 

evaluator- Obsolescence, would simply ,requir,e a greater emphasis on 

periodic audits W programs. And' what would those audits look like? In 

large parf that would need to be determined by ^the information requi remei^ts 

of the agency asking for the audit. I believe that traditional notions 

of^ /he audit that are tied to^ traditional notions about t}|e evaluation 

might very well go^ by the wayside and be replaced by audit's that are 

quite different than vje novj conduct them. 

Another loose end, given this different conception of evaluation, 
is that there VN/ould be the necessity for restructuring report requirements 
of externally funded programs. If the focus is on self-eVaJ^ the 
implication is that whjan the project staff is engaged in the process of 
systematic sel f-ev$luation using procedures that they understand, tnen 
they are most likely t^^ use that evaluative "information in making program 
changes that are meaningful. 

Some might maintain that the reporting procedures, program plan, 
and other such things of a particular externally funded program (state 
or'fedGral) are systematic are conceptual ly sequentTal, and do provide 
* a mechanism for program' understanding. Let me not -quarrel with these 



assertions and let me agree th^t each of Chese documents, reports,^^^ 
format, etc. was conceived qf in a systematic way and.^represents an 
appropriate problem solving/program improvement methodology. The problj^ 
becomes the translation of these procedures to people in the fi^ld wh( 
are-faced y/ith the necessity of filling them out.' Unfortunately most 
people In school districts do not vievj the procedure as a systematic . 
sequence of events to be aocompl ished -in order. to obtain program under- 
standi^ng and ultimately program imVovement. Instead, the^see a steady/ 
parade of forms filing faithfully past the^ir desk, -and demanding to be 
handled properly. 'Ofie learns the sequence, learns the things that one 
has to do, and 'consequently the process become^ viewed by project 
personnel as something external to them that they comply with but a^e 
not committed to, which is^"the State" but is not "us." 

If the emphasis is to be on self-evaluation then the reporting 
sch^e should in' some way demonstrate that projects are engaging in 
self-monitoring, self-clarifying, self-acquisition of information, and 
self-correction. In short, self-evaluation is v;hat is hopefully attained 
- And what the external agency should vant to have reported to/t is not 
how vjel^'the program did, nor how v^ell the program was implemented, but 
instead is how well the prc^ject began the process of understanding what 
it hoped to accomplish, how well the project identified and recognized 
aspects of the program that they vjere uncomfortable with and how i«ell 

.■t 

the'^roject responded to its own feedback information in making self- 
corrections. This kind of evaluation report focusing on the power of 
self-evaluation, must be oriented towards capturing evidence that the 
self-evaluation process is taking place. TWs.lTind of sel f-eval^uati on 
reporting is far more inlportant th^n the reporting of the outcomes of 
the proqrani^to external agencies. 



/ 

,/r And that my message ftr today. 



Applajjse 

^ Remember when I began this speech I forbade j^ou from applauding 
wildly, standing 'up,^stamping your feet, etc.? Now, let me remove that 
.prohibition and close with the- comment that "I propose that we create a 
ifew position of information specialist and do away with^the* role of 
evaluator." . ^ . ' ' . ^ 
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